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ABSTRACT 



Image data is processed by a low level feature detection 
processor that extracts low level features from an im- 
age, This is accomplished by converting a matrix of 
image data into a matrix of orthogonal icons that sym- 
bolically represent the image scene using a predeter- 
mined set of attributes. The orthogonal icons serve as 
the basis of processing by means of a high level graph 
matching processor which employs symbolic scene 
segmentation, description, and recognition processing 
that is performed subsequent to the low level feature 
detection. This processing generates attribute graphs 
representative of target objects present in the image 
scene. High level graph matching compares predeter- 
mined attributed reference graphs to the sensed graphs 
to produce a best common subgraph between the two 
based on the degree of similarity between the two 
graphs. The high level graph matching generates a 
recognition decision based on the value of the degree of 
similarity and a predetermined threshold. The output of 
the high level graph matching provides data from 
which a target aimpoint is determined, and this aimpoint 
is coupled as an input to a missile guidance system that 
tracks identified targets. 

16 Claims, 6 Drawing Sheets 
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Region object attributes .„ 
Region Boundary 




— Original Region Boundary 
— — Convex Hull 

Perimeter = d1 +d2+d3+d4+d5+d6 

Area = (standard computation of polygon area) 

2 (midpoint of segment) x (length of segment) 

Centroid (X,Y) = numoeror segments 

Similar equations apply for the convex hull polygon 
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reference numerals designate like structural elements, 

PATTERN RECOGNITION APPARATUS and in which: 

UTILIZING AREA LINKING AND REGION FIG. 1 is a block diagram of a scene recognition 

GROWTH TECHNIQUES system employing low and high level feature detection 

5 to identify and track targets in accordance with the 

CROSS REFERENCE TO RELATED principles of the present invention; 

APPLICATIONS FIG. 2 is a detailed functional block diagram of the 

The present application is related to patent applica- system of FIG. 1; 

tions Ser. No. 514,778 filed on Apr. 25, 1990, entitled FIGS. 3c-3/ show the processing performed to 

"Improved Data Compression System and Method," 10 achieve low level feature detection; 

and Ser. No. 514,779 filed on Apr. 25, 1990, entitled FIGS, ta and 4* show the results of flat linking and 

"Improved Data Decompression System and Method," region growth processing performed in the scene seg- 

whose teachings are incorporated herein by reference. mentation section of FIG. 2; 

uArirrpnTTMn i < F1GS - show the results of boundar y formation 

bACKUKUUJNU 15 ^ feature extr action processing performed in 

The present invention relates generally to scene rec- tne scene segmentation section of FIG. 2; 
ognition systems and methods, and more particularly, to FIGS. 6 and 7 show attributes computed for region 
a scene recognition system and method that employs ^ ribbon objects, respectively, in object formation 
low and high level feature detection to identify and processing performed in the scene segmentation section 
track targets. 20 0 f piG. 2; 

Modern missile scene recognition systems employ FIGS the transitional steps in generating 

specialized signal processing architectures and algo- m attributcd sensed graph; and 
rithms that are designed to quickly and efficiently de- 9q and 9b shQW fl reference ^ an attr ib- 

tect the presence of particular target objects such as reference respcctively; 

building trucks tanks, and ships, and the like that are 25 * h determined 

located m the field of view of the missi e Oo^og, matchi ^ Jac recognition 

more sophisticated designs are always in demand that v * » / r , * 

^•od C oS identify ° rCl3SSify UrgCtsinaVerySh0rt ^SgS £ and Ub show a reference scene and a 
pen0 ° ' 30 sensed scene having the respective aimpoints designated 

SUMMARY OF THE INVENTION therein. 

The present invention comprises a scene recognition DETAILED DESCRIPTION 

system and method for use with a missile guidance and 

tracking system, that employs low and high level fea- Referring to the drawings, FIG. 1 is a block diagram 
rure detection to identify and track targets. The system 35 of a scene recognition system employing low and high 
employs any conventional imaging sensor, such as an level feature detection to identify and track targets in 
infrared sensor, a millimeter wave or synthetic aperture accordance with the principles of the present invention, 
radar, or sonar, for example, to image the scene. The The system comprises a low level feature detector 11 
output of the sensor (the image) is processed by a low that is adapted to receive image data derived from an 
level feature detection processor that extracts low level 40 imaging sensor 9, such as an infrared sensor, a television 
features from the image. This is accomplished by con- sensor, or a radar, for example. The low level feature 
verting a matrix of sensor data (image data) into a ma- detector 11 is coupled to a high level graph matching 
trix of orthogonal icons that symbolically represent the processor 12 that is adapted to process icons represents- 
image using a predetermined set of attributes. The or- ^ve Q f features contained in the image scene that are 
thogonal icons serve as the basis for processing by 45 generated by the low level feature detector 11. Aim- 
means of a high level graph matching processor which information is generated and used by a missile 
employs symbolic scene segmentation, description, and navigation system 24 to steer a missile toward a desired 
recognition processing that is performed subsequent to target 

the low level feature detection. The process generates ^ M h levd graph ma tching processor 12 includes 
attributed graphs representative of target objects pres- 50 a synt hesi2er 13, graph matcher 

ent in the image. The high level graph matching proces- u ^ ^ mt estimator 15. A reference graph stor- 
sor compares predetermined attributed reference me Jtf ^ ^ t0 tfae h matching pro . 

graphs to the sensed graphs ; to produce a best common ^ ^ ^ store ^ 

subgraph 1 between the two, ^^f^j^ „ representing expected target objects that are present in 
lanty between the two graphs. 55 ^ image scenf-The reference graphs include graphs 

predetennined threshold. The output of the high level bodies, for example. ^ 

kraph matching processor provides data from which an j The low level feature detector 11 is descried in 
aimpoint is determined. The aimpoint is coupled as an 60 detail in U.S. patent applications Ser. No. 514,779 filed 
input to the missile guidance system that tracks an iden- on Apr. 25, 1990, entitled Improved Data Decompres- 
sed target sion Svstem m< * Method," whose teachings arc incor- 
porated herein by reference. In summary, however, the 
BRIEF DESCRIPTION OF THE DRAWINGS i ow i eve i feature detector 11 converts a matrix of sensor 
The various features and advantages of the present 65 data (image data) into a matrix of orthogonal icons that 
invention may be more readily understood with refer- symbolically represent the imaged scene through the 
ence to the following detailed description taken in con- use of a set of attributes. These are discussed in more 
junction with the accompanying drawings, wherein like detail below with reference to FIGS. 3-11. 
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The processing performed by the scene recognition matrix of orthogonal icons that represent the imaged 
system shown in FIG. 1 employs a series of transforma- scene symbolically via a set of attributes. These orthog- 
tions that converts image information into pregressively onal icons serve as the basis for performing symbolic 
more compressed and abstract forms. The first transfer- scene segmentation, description, and recognition pro- 
nation performed by the low level feature detector 11 5 cessor shown in FIG. 2. FIG. 3a represents an image of 
converts the sensor image, which is an array of numbers a house 41 whose sides 42, 43 and roof sections 44, 45 
describing the intensity at each picture position, into a have different shading due to the orientation of the sun 
more compact array of icons with attributes that de- relative thereto. Each face of the house 41 is identified 
scribed the essential intensity distributions of 10x10 by a different texture (shading) identified in FIG. 3a. 
pixel blocks of image data. 10 The image of the house 41 is an array of numbers repre- 

The second transformation implemented in the high sentative of different intensities associated with each 

level graph matching processor 12 links the icon array pixel of data. 

into separate regions of nearly uniform intensity and The image data representing the house 41 is pro- 
identifies linear boundaries between regions. A special cessed by a 10 X 10 block window 46 as is roughly illus- 
case of a region that is long and narrow with roughly 15 trated in FIG. 36 which generates pixel data illustrated 
parallel sides is identified as a ribbon. This transforma- in FIG. 3c. The pixel data of FIG. 3c is generated in the 
tion results in a list of the objects formed, various attri- low level feature detector 11. This pixel data contains 
butes of the objects themselves, and relationships be- orthogonal icons. Each pixel in FIG. 3c is shaded in 
tween the objects. This information is encoded in a accordance with the shade it sees in FIG. 36. The 
structure identified as an attributed graph. 20 blocks or cells of data shown in FIG. 3c are generated 

The graph matcher 14 compares the graph structure in the low level feature detector 11. 
derived from the sensor image with a previously stored In FIG. 3d lines are formed, and in FIG. 3* regions 
graph derived earlier or from reconnaissance informs- are formed from the shaded pixels in FIG. 3c. The lines 
tion, for example. If a high degree of correlation is are generated by gradient based segmentation process- 
found, the scene described by the reference graph is 25 ing 33 in the scene segmentation processor 21 shown in 
declared to be the same as the scene imaged by the FIG. 2, while the regions are generated by the intensity 
sensor. Given a match, the aimpoint estimator 15 associ- based segmentation processing 32 in the scene segmen- 
ates the aimpoint given apriori in the reference graph to tation processor 21 shown in FIG. 2. The regions are 
the sensed graph, and this information is provided as an formed using flat linking and gradient linking and the 
input to the missile navigation and guidance system. 30 region shapes are determined. The line and region infor- 

In a classical graphing application, the general prob- mation is then processed by gradient/intensity object 

lem of correlating two graphs is N-P complete, in that a merging 34 and the graph synthesis processing 35 

linear increase in the number of nodes requires an expo- shown in FIG. 2. in order to form regions and ribbons, 

nential increase in the search, or number of compari- and a graph is synthesized, as illustrated in FIG. 3/, 

sons, required to match the graphs. In the present inven- 35 which is a graph representation of the house 41 of FIG. 

tion, this problem is solved by overloading the sensed 3a. An aimpoint is shown in FIG. 3/that is derived from 

and reference graphs with unique attributes that signifi- the information comprising the reference graph 36 

cantly reduces the search space and permits rapid, real shown in FIG. 2. 

time searching to be achieved. The flat linking and region growth processing is part 

FIG. 2 is a detailed functional block diagram of the 40 of the scene segmentation processing 21 of FIG. 2. This 

system of FIG. 1. Scene segmentation processing 21 flat linking and region growth processing is comprised 

comprises low level feature description processing, of two subprocesses. The flat linking process groups 

intensity and gradient based segmentation processing low level feature discrimination orthogonal icons of 

32, 33 and gradient/intensity object merging 34, all type FLAT into homogeneous intensity flat regions 

performed by the low level feature detector 11 (FIG. 1). 45 using a relaxation-based algorithm. The result is a set of 

Output signals from the scene segmentation processing regions comprised of FLAT icons and described by 

21 are coupled to scene description processing 22 that their area (number of constituent FLAT icons), inten- 

comprises graph synthesis processing 35 that produces a sity (average intensity of the constituent FLAT icons), 

sensed graph of objects present in the image. Reference and a list of the constituent FLAT icons. More particu- 

graphs 36 are stored that comprise graphs representing 50 larly, groups of orthogonal icons having homogeneous 

vehicles, buildings, landmarks, and water bodies, for intensity regions are formed to generate a set of regions 

example. These graphs are prepared from video data having a block resolution boundary. The groups of 

gathered during reconnaissance flights, for example, orthogonal icons are comprised of homogeneous inten- 

which provide mission planning information. Target sity icons described by their area, including the number 

objects that are present in the video data are processed 55 of constituent icons having homogeneous intensity, the 

by the present invention to generate the reference average intensity of the constituent homogeneous inten- 

graphs which are ultimately used as comparative data sity icons* and a list of the constituent homogeneous 

from which' target objects are identified and selected intensity icons, region growth processing appends adja- 

during operational use of the invention. cent orthogonal icons having an intensity gradient 

Scene recognition processing 23 employs the graphs 60 thereacross to provide a feature-resolution boundary, 

generated by the graph synthesis processing 35 and the The region growth process appends adjacent orthog- 

reference graphs 36 by means of graph matching pro- onal icons of type gradient (non FLAT) information to 

cessing 37 and scene transformation processing 38 the grown flat regions thus providing a feature-resolu- 

which generate the aimpoint estimate that is fed to the tion boundary as opposed to the 10 X 10 block resolu- 

navigation system 24. 65 tion boundary provided by the flat linking process. The 

FIGS. 3a-3/ show the processing performed to method employed is as follows. (1) Being with a flat 

achieve low level feature discrimination. This process region provided by the flat linking process. The region 

converts a matrix of sensor data (image data) into a is comprised solely of FLAT icons and grows up to 
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flanking gradient icons which stop the growth. (2) Con- 
sider the gradient icons flanking the region. Each gradi- 
ent icon is described by a bi-intensity model of which 
one of the intensities are adjacent to the flat region. If 
this intensity is similar to the intensity of the flat region 
extend the region boudary to include the portion of the 
gradient icon covered by the intensity. (3) Repeat for all 
flat regions. Flat linking assigns a single region number 
to every FLAT icon. Region growth assigns multiple 



region numbers to gradient icons. Gradient icons of 10 35 in FIG. 2. 



linking and region growing processes thus producing 
region objects 76-78 (FIGS. 8a, Sb). These ribbon and 
region objects become the nodes of the attributed 
sensed graph. The symbolic descriptions serve as the 
attributes. FIGS. 6 and 7 show attributes computed for 
region and ribbon objects 75-78, shown in FIGS. 80 and 
8/>, respectively, in object formation processing per- 
formed by the scene segmentation process 21 of FIG. 2, 
and more particularly by the graph synthesis processing 



type EDGE and CORNER are assigned two region 
numbers, those of type RIBBON are assigned three 
region numbers. Gradient icons of type SPOT are not 
considered. The result of this process is a set of homoge- 
neous intensity regions consisting of FLAT icons and IS 
partial gradient icons. This is described in detail in the 
"Improved Data Decompression System and Method" 
patent application cited above. 

FIGS. 4a and 4b show the results of flat linking and 
region growth processing performed by the scene seg- 20 
mentation processing 21 of FIG. 2. In particular this 
processing is performed by intensity based segmenta- 
tion processing 32, gradient based segmentation pro- 
cessing 33, and the gradient/intensity object merging 34 
processes shown in FIG. 2. In FIGS. 4a and 4b, the 25 
orthogonal icons containing a single shade of gray rep- 
resent FLAT icons 50, 51. Those that contain two 
shades of gray (along the diagonal) represent EDGE 
icons 52. Regions are comprised of both FLAT and 
gradient (non FLAT) icons. EDGE icons 52, as well as 30 
all other gradient icons, are associated with multiple 
regions, that is, they are assigned multiple region num- 
bers. In FIGS. 4a and 46, each of the EDGE icons 52 
are assigned two region numbers, 1 and 2. This provides 
us with boundaries that are accurate to the gradient 35 
features detected by the low level feature discrimination 
process. 

Boundary formation and linear feature extraction 
processing is performed by the scene segmentation pro 



FIG. 6 shows region object attributes and computa- 
tions for a region boundary and a convex hull using the 
equations shown in the drawing. With reference to 
FIG. 7, it shows the ribbon object attributes, including 
ribbon length in FIG. la, ribbon intensity in FIG. lb, 
polarity in FIG. 7c, and ribbon orientation in FIG. Id. 
The arrows shown in FIGS, lb and 7c are indicative of 
the orientation of the line segments that constitute the 
ribbon. 

For region objects, the process computes the area, 
perimeter, and convex hull attributes. For ribbon ob- 
jects, the process searches through a line table looking 
for pairs of lines that: (1) differ in orientation by 180 
degrees, (2) are in close proximity to each other, (3) are 
flanked by similar intensities, and (4) do not enclose 
another line segment that is parallel to either line. When 
a pair of lines fitting these constraints is found the rib- 
bon attributes for them are computed. The attributes for 
ribbon objects include: (1) intensity of the ribbon, (2) 
polarity of the ribbon (hg* 1 * on dark or dark on light), 
(3) the width of the ribbon (distance between two lines), 
and (4) the orientation of the ribbon. 

The graph synthesis process and its associated graph 
language computes relational descriptions between the 
region and ribbon objects 75-78 (FIGS. 8a, *b) pro- 
vided by the object formation process and formulates 
the attributed sensed graph structure from the region 
and ribbon objects and their relationships. The relation- 
ships between every region/region, region/ribbon, and 



cessing of FIG. 2. This processing is comprised of three 40 ribbon/ribbon pair are computed. The relationships are 



sub-processes. The gradient boundaries of each region 
60-63 (FIG. 5a) are traversed forming a gradient chain 
around each region. The gradient chains for each region 
are analyzed for linear segments 64-68 (FIG. Sb) and 
having pointers inserted into the chains at the beginning 45 
and end of each linear segment 64-68. The linear seg- 
ments 64-68 are analyzed for the presence of adjacent 
segments related to two or more regions that form line 
segments. The results of these processes include bound- 



of type spatial (proximity to one another) or compara- 
tive (attribute comparisons). Graphs, nodes, links be- 
tween- nodes, and graph formation and processing is 
well known in the art and will not be described in detail 
herein. One skilled in the art of graph processing should 
be able to derive the graphs described herein given a 
knowledge of the attributes that are to be referenced 
and the specification of the graphs described herein. 
In formulating the attributed sensed graph, the region 



ary descriptions for each of the regions 60-63 formu- 50 and ribbon objects are placed at graph nodes, one object 



lated by the flat linking and region growth processes, 
and linear segments 70-73 (FIG. 5c) represented by 
length, orientation, and end point coordinates. 

FIGS. 5a-5c show the results of boundary formation 
and linear feature extraction processing performed in 55 
the scene segmentation processing 21 of FIG. 2. FIG. 
Sa shows the regions 60-*3 derived by the flat linking 
and region growth process. FIG. Sb shows the bound- 
ary descriptions of each of the regions 60-63 as derived 
by the boundary formation process. FIG. 5c shows the 60 
line segments 70-73 derived by the linear feature extrac- 
tion process. 

The object formation process is comprised of two 
subprocesses. The first forms ribbon objects 75 from the 
linear features provided by the gradient boundary and 65 
linear feature extraction processes and creates symbolic 
descriptions of these objects. The second creates sym- 
bolic descriptions of the regions provided by the flat 



per node, along with their descriptive attributes. The 
relationships between each pair of objects are placed at 
the graph arcs (links) along with their attributes. With 
this scheme, a fully connected attributed graph is for- 
mulated which symbolically represents the original 
imaged scene. 

FIGS. 8<?-8c show the transitional steps in generating 
an attributed sensed graph 80 in the graph synthesis 
process 35 (FIG. 2). The graph 80 is comprised of nodes 
81-84, that symbolically represent objects within the 
scene, and node attributes that describe the objects, and 
arcs (or links) that symbolically represent the relation- 
ships between the objects within the scene, and link 
attributes, that describe the relationships between the 
objects. More particularly, node 1 is representative of 
region 1 object 76, node 2 is representative of region 2 
object 77, node 3 is representative of region 3 object 78, 
and node 4 is representative of the ribbon object 75. The 
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various links between the ribbon and region objects 
75-78 are shown as 1 link 2, 2 link 4, 1 link 4, 1 link 3, 
2 link 3, and 3 link 4. The links include such parameters 
as the fact that the region 1 object 76 is above and adja- 
cent the region 2 object 77, that the region 1 object 76 
is left of the region 3 object 78, that the region 2 object 
77 is left and adjacent to the ribbon object 75, and so 
forth. 

Reference graph processing entails the same func- 
tions as the object formation and graph synthesis pro- 
cesses with two differences. First, it is performed prior 
to the capture of the sensed image and subsequent scene 
segmentation and description. Second, it receives its 
inputs from a human operator by way of graphical input 
devices rather than from autonomous segmentation 
processes. 

FIGS. 9a and 9b show a reference scene 90 and an 
attributed reference graph 91, respectively. FIG. 9a 
shows the reference scene 90 comprised of region 1 and 
region 2, the ribbon and a desired aimpoint 94. The 20 
attributed reference graph 91 shown in FIG. 9* is com- 
prised of nodes and links substantially the same as is 
shown in FIG. 8c, The nodes and links comprise the 
same types of relationships described with reference to 
FIG. 8<r. 

The graph matching process 37 (FIG. 2) compares 
attributed reference and attributed sensed graphs, gen- 
erates a best common subgraph between the two based 
on the degree of similarity between the two graphs 
(confidence number), and generates a recognition deci- 
sion based on the value of the confidence number and a 
predetermined threshold. Feasibility is determined by 
the degree of match between the node and are attributes 
of each of the graphs. A heuristic procedure is included 
to ignore paths that cannot produce a confidence num- 
ber larger than the predetermined threshold. Also, 
paths which lead to ambiguous solutions, that is, solu- 
tions tha match a single object in one graph to multiple 
objects in the other are, ignored. Finally, path that do 
not preserve the spatial relationships between the ob- 40 
jects as they appeared in the original scene are ignored. 

FIG. 10 shows a best common subgraph 92 deter- 
mined by the graph matching section of the scene rec- 
ognition processing 23 of FIG. 2. The graph shown in 
FIG. 8c is matched against the graph shown in FIG. 9b 45 
which generates the common subgraph 92 of FIG. 10. 
This is performed in a conventional manner well-known 
in the graph processing art and will not be described in 
detail herein. 

FIGS, 11a and Mb show a reference scene 90 and a 50 
sensed scene 93 having their respective aimpoints 94, 95 
designated therein. More particularly, FIGS, lie and 
lib depict one possible scenario in which an object is 
detected in the sensed scene (region 3) but is not desig- 
nated in the reference graph. An appropriate aimpoint is 
designated in spite of this difference as well as others 
including object additions or omissions, object size and 
shape differences, and changes due to imaging condi- 
tions. The aimpoint 94 in the reference scene 90 is in- 
cluded in the attributed reference graph and is indica- 60 
tive of a desired target aiming location. This aimpoint is 
"transferred'* to the sensed scene as the aimpoint that 
the missile should be directed at. This information is 
transferred to the missile navigation system for its use. 

Thus there has been described a new and improved 65 
scene recognition system and method that employs low 
and high level feature detection to identify and track 
targets. It is to be understood that the above-described 
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embodiment is merely illustrative of some of the many 
specific embodiments which represent applications of 
the principles of the present invention. Clearly, numer- 
ous and other arrangements can be readily devised by 
those skilled in the art without departing from the scope 
of the invention. 
What is claimed is: 

1. A scene recognition system employing low and 
high level detection to identify and track targets located 
in an image scene and a missile guidance system adapted 
to steer a missile toward a desired target, said system 
comprising: 

low level feature detection processor means for pro- 
cessing image data derived from and representative 
of an imaged scene, and for extracting features 
from the imaged scene by converting the image 
data into a matrix of orthogonal icons that symboli- 
cally represent the image using a predetermined set 
of attributes, said low level feature detection pro- 
cessor means comprising: 

a) flat linking processing means for forming groups 
of orthogonal icons having homogeneous inten- 
sity regions to generate a set of regions having a 
block resolution boundary and that are com- 
prised of homogeneous intensity icons described 
by their area, comprising the number of constitu- 
ent icons having homogeneous intensity, the 
intensity, comprising the average intensity of the 
constituent homogeneous intensity icons, and a 
list of the constituent homogeneous intensity 
icons; and 

b) region growth processing means coupled to the 
flat linking processing means for appending adja- 
cent orthogonal icons having an intensity gradi- 
ent thereacross to provide a feature-resolution 
boundary; 

graph synthesis processor means coupled to the low 
level feature detection processing means for pro- 
cessing the orthogonal icons to generate predeter- 
mined objects representative of objects that are in 
the scene, and for computing relational descrip- 
tions between the objects to form an attributed 
sensed graph from the objects and their relation- 
ships as described by their attributes, and where- 
upon the objects are placed at graph nodes, one 
object per node, along with their descriptive attri- 
butes, and wherein the relationships between ob- 
ject pairs are placed at graph links along with their 
attributes, and whereupon a fully connected attrib- 
uted graph is formulated which symbolically repre- 
sents the image scene; . 
reference graph storage means coupled to the graph 
synthesis processing means for storing predeter- 
mined reference graphs representative of identifia- 
ble targets of interest that are expected to be pres- 
ent in the data comprising the image; and 
graph matching processing means coupled to the 
graph synthesis processing means for comparing 
predetermined attributed reference graphs to the 
sensed graphs to produce an object recognition 
decision based on the value of the degree of similar- 
ity between the attributed reference graphs to the 
sensed graphs and a predetermined threshold, and 
for providing an output signal that is determinative 
of a target aimpoint, which output signal is coupled 
as an input to the missile guidance system to pro- 
vide a guidance signal that is adapted to steer the 
missile toward the identified target. 
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2. The system of claim 1 wherein the low level fea- 
ture detection processing means further comprises: 

boundary formation and linear feature extraction 
processing means coupled to the region growth 
processing means for (1) traversing the gradient 
boundaries of each region to form a gradient chain 
around each region, (2) analyzing the gradient 
chains for each region for linear segments by means 
of pointers inserted into the chains at the beginning 
and end of each linear segment, and (3) analyzing 
the linear segments for the joining of segments 
related to two or more regions to form linear seg- 
ments, which generates boundary descriptions for 
each of the regions formulated by the flat linking 
and region growth processing means and linear 15 
segments represented by length, orientation, and 
end point coordinates. 

3. The system of claim 2 wherein the low level fea- 
ture detection processing means further comprises: 

object formation processing means coupled to the 20 
boundary formation and linear feature extraction 
processing means for forming ribbon objects from 
the linear features provided by the gradient bound- 
ary and linear feature extraction processing means 
and for creating symbolic descriptions of these 25 
objects, by creating symbolic descriptions of the 
regions provided by the flat linking and region 
growing processing means to produce region ob- 
jects that define nodes of an attributed sensed 
graph, and whose symbolic descriptions comprise 30 
the attributes; 

whereby for region objects, the object formation 
processing means computes the area, perimeter, 
and convex hull attributes, and for the ribbon ob- 
jects, the object formation processing means 35 
searches through a line table looking for pairs of 
lines that: (1) differ in orientation by 180 degrees, 
(2) are in close proximity to each other, (3) are 
flanked by similar intensities, (4) do not enclose 
another line segment that is parallel to either line, 
and when a pair of lines fitting these constraints is 
found the ribbon attributes for them are computed, 
and wherein the attributes for ribbon objects in- 
clude: (1) intensity of the ribbon, (2) polarity of the 
ribbon, meaning light on dark or dark on light, (3) 45 
the width of the ribbon, meaning the distance be- 
tween the two lines, and (4) the orientation of the 
ribbon. 

4. The system of claim 3 wherein the graph matching 
processing means compares predetermined attributed 
reference graphs to the sensed graphs to produce a best 
common subgraph between the two based on the degree 
of similarity between the two graphs, and generates a 
recognition decision based on the value of the degree of 
similarity and a predetermined threshold. 

5. The system of claim 4 wherein the graph matching 
processing means utilizes a heuristically directed depth- 
first search technique to evaluate feasible matches be- 
tween nodes and arc attributes of the attributed refer- 
ence and sensed graphs. 

6. The system of claim 5 wherein the graph matching 
processing means determines feasibility by the degree of 
match between the node and arc attributes of each of 
the graphs, and a heuristic procedure is included to 
ignore paths that cannot produce a degree of similarity 65 
larger than the predetermined threshold. 

7. The system of claim 6 wherein the paths which 
lead to ambiguous solutions, comprising solutions that 



40 



50 



55 



60 



match a single object in one graph to multiple objects in 
the other, are ignored, and wherein paths that do not 
preserve the spatial relationships between the objects as 
they appeared in the image are ignored. 

8. A scene recognition system employing low and 
high level feature detection to identify and track targets 
located in an imaged scene and a missile guidance sys- 
tem adapted to steer the missile toward a desired target, 
said system comprising: 
low level feature detection processing means adapted 
to process image data derived from and representa- 
tive of an image scene, for extracting low level 
features from the image by converting the image 
data into a matrix of orthogonal icons that symboli- 
cally represent the image scene using a predeter- 
mined set of attributes; 
graph synthesis processing means coupled to the low 
level feature detection processing means process- 
ing the orthogonal icons and for computing rela- 
tional descriptions between the region and ribbon 
objects provided by the object formation process- . 
ing means to form die attributed sensed graph from 
the region and ribbon objects and their relation- 
ships, and whereupon the region and ribbon ob- 
jects are placed at graph nodes, one object per 
node, along .with their descriptive attributes, the 
relationships between each pair of objects are 
placed at the graph links along with their attri- 
butes, whereupon, a fully connected attributed 
graph is formulated which symbolically represents 
the image; 

reference graph storage means coupled to the graph 
synthesis processing means for storing predeter- 
mined reference graphs representative of identifia- 
ble targets of interest that are expected to be pres- 
ent in the data comprising the image; and 
graph matching processing means coupled to the 
graph synthesis processing means for comparing 
predetermined attributed reference graphs to the 
sensed graphs to produce an object recognition 
decision, which produces a best common subgraph 
between the two based on the degree of similarity 
between the two graphs, for generating a recogni- 
tion decision based on the value of the degree of 
similarity and a predetermined threshold, and for 
providing an output signal that is determinative of 
a target aim point, which output signal is coupled as 
an input to the missile guidance system to provide 
a guidance signal that is adapted to steer the missile 
toward the identified target. 
$. A scene recognition system employing low and 
high level feature detection to identify and track targets 
located in an image scene, said system comprising: 
low level feature detection processing means adapted 
to process image data derived from and representa- 
tive of an image scene, for extracting low level 
features from the image scene by converting the 
image data into a matrix of orthogonal icons that 
symbolically represent the image using a predeter- 
mined set of attributes; 
flat linking processing means coupled to the low level 
feature detection processing means for forming 
groups of orthogonal icons having homogeneous 
intensity regions by means of a relaxation-based 
algorithm to generate a set of regions having a 
block resolution boundary and that are comprised 
of homogeneous intensity icons described by their 
area, comprising the number of constituent icons 
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having homogeneous intensity, the intensity, com- 
prising the average intensity of the constituent 
homogeneous intensity icons, and a list of the con- 
stituent homogeneous intensity icons; 
region growth processing means coupled to the flat 5 
linking processing means for appending adjacent 
orthogonal icons having an intensity gradient 
thereacross to provide a feature-resolution bound- 
ary; 

boundary formation and linear feature extraction 10 
processing means coupled to the region growth 
processing means for (1) traversing the gradient 
boundaries of each region to form a gradient chain 
around each region, (2) analyzing the gradient 
chains for each region for linear segments by means 15 
of pointers inserted into the chains at the beginning 
and end of each linear segment, and (3) analyzing 
the linear segments for the joining of segments 
related to two or more regions to form linear seg- 
ments, which generates boundary descriptions for 20 
each of the regions formulated by the flat linking 
and region growth processing means and linear 
segments represented by length, orientation, and 
end point coordinates; 

object formation processing means coupled to the 25 
boundary formation and linear feature extraction 
processing means for forming ribbon objects from 
the linear features provided by the gradient bound- 
ary and linear feature extraction processing means 
and for creating symbolic descriptions of these 30 
objects, by creating symbolic descriptions of the 
regions provided by the flat linking and region 
growing processing means to produce region ob- 
jects that define nodes of an attributed sensed 
graph, and whose symbolic desctiptions comprise 35 
the attributes; 

whereby for region objects, the object formation 
processing means computes the area, perimeter, 
and convex hull attributes, and for the ribbon ob- 
jects, the object formation processing means 40 
searches through a line table looking for pairs of 
lines that: (1) differ in orientation by 180 degrees, 
(2) are in close proximity to each other, (3) are 
flanked by similar intensities, (4) do not enclose 
another line segment that is parallel to either line, 45 
and when a pair of lines fitting these constraints is 
found the ribbon attributes for them are computed, 
and wherein the attributes for ribbon objects in- 
clude: (1) intensity of the ribbon, (2) polarity of the 
ribbon, meaning light on dark or dark on light* (3) 50 
the width of the ribbon, meaning the distance be- 
tween the two lines, and (4) the orientation of the 
ribbon; 

graph synthesis processing means coupled to the low 
level feature detection processing means process- 55 
ing the orthogonal icons and for computing rela- 
tional descriptions between the region and ribbon 
objects 'provided by the object formation process- 
ing means to form the attributed sensed graph from 
the region and ribbon objects and their relation- 60 
ships, and whereupon the region and ribbon ob- 
jects are placed at graph nodes, one object per 
node, along with their descriptive attributes, the 
relationships between each pair of objects are 
placed at the graph links along with their attri- 65 
butes, whereupon, a fully connected attributed 
graph is formulated which symbolically represents 
the imaged scene; 
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reference graph storage means coupled to the graph 
synthesis processing means for storing predeter- 
mined reference graphs representative of identifia- 
ble targets of interest that are expected to be pres- 
ent in the data comprising the image scene; and 

graph matching processing means coupled to the 
graph synthesis processing means for comparing 
predetermined attributed reference graphs to the 
sensed graphs to produce a best common subgraph 
between the two based on the degree of similarity 
between the two graphs, and for generating a rec- 
ognition decision based on the value of the degree 
of similarity and a predetermined threshold, and 
wherein the graph matching processing means 
utilizes a heuristically directed depth-first search 
technique to evaluate feasible matches between the 
nodes and arcs of the attributed reference and 
sensed graphs, wherein feasibility is determined by 
the degree of match between the node and arc 
attributes of each of the graphs, and a heuristic 
procedure is included to ignore paths of the tree 
that cannot possibly produce a degree of similarity 
larger than the predetermined threshold, and 
wherein the paths of the tree which lead to ambigu- 
ous solutions, wherein solutions that match a single 
object in one graph to multiple objects in the other, 
are ignored, and wherein paths of the tree that do 
not preserve the spatial relationships between the 
objects as they appeared in the original scene are 
ignored. 

10. A method for use with a missile guidance system 
to track targets located in an imaged scene, said method 
comprising the steps of: 
processing image data derived from and representa- 
tive of an imaged scene to form groups of orthogo- 
nal icons, having homogeneous intensity regions, 
by means of a relaxation-based algorithm to gener- 
ate a set of regions having a block resolution 
boundary and that are comprised of homogeneous 
intensity icons described by their area, comprising 
the number of constituent icons having homogene- 
ous intensity, the intensity comprising the average 
intensity of the constituent homogeneous intensity 
icons, and a list of the constituent homogeneous 
intensity icons; 
appending adjacent orthogonal icons having an inten- 
sity gradient thereacross to provide a feature-reso- 
lution boundary; 
processing the orthogonal icons to form an attributed 
sensed graph from region and ribbon objects com- 
prising the orthogonal icons and their relationships, 
and whereupon the region and ribbon objects are 
placed at graph nodes, one object per node, along 
with their descriptive attributes, the relationships 
between each pair of objects are placed at the 
graph links along with their attributes, whereupon, 
a fully connected attributed graph is formulated 
which symbolically represents the image scene; 
storing predetermined reference graphs representa- 
tive of identifiable targets of interest that are ex- 
pected to be present in the data comprising the 
image scene; and 
comparing predetermined reference graphs to the 
attributed sensed graphs to produce an object rec- 
ognition decision, and providing an output signal 
that is determinative of a target aimpoint, which 
output signal is coupled as an input to the missile 
guidance system to provide a guidance signal that 



03/29/2004, EAST Version: 1.4.1 



5,093, 

13 

is adapted to steer a missile toward the identified 
target. 

11. The method of claim 10 wherein the step of pro- 
cessing image data comprises: 

forming regions by (1) traversing the gradient bound- 5 
aries of each region to form a gradient chain 
around each region, (2) analyzing the gradient 
chains for each region for linear segments by means 
of pointers inserted into the chains at the beginning 
and end of each linear segment, and (3) analyzing 10 
the linear segments for the joining of segments 
related to two or more regions to form linear seg- 
ments, which generates boudary descriptions for 
each of the regions formulated by the flat linking 
and region growth processing means and linear 15 
segments represented by length, orientation, and 
end point coordinates. 

12. The method of claim 11 wherein the step of pro- 
cessing image data comprises: 

forming ribbon objects from the linear features pro- 20 
vided by the gradient boundary and linear feature 
extraction processing step and for creating sym- 
bolic descriptions of these objects, by creating 
symbolic descriptions of the regions provided by 
the flat linking and region growing processing 25 
means to produce region objects that define nodes 
of an attributed sensed graph, and whose symbolic 
descriptions comprise the attributes; 

whereby for region objects, the area, perimeter, and 
convex hull attributes are determined, and for the 30 
ribbon objects, a line table is searched looking for 
pairs of lines that: (1) differ in orientation by 180 
degrees, (2) are in close proximity to each other, (3) 
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are flanked by similar intensities, (4) do not enclose 
another line segment that is parallel to either line, 
and when a pair of lines fitting these constraints is 
found the ribbon attributes for then are computed, 
and wherein the attributes for ribbon objects in- 
clude: (1) intensity of the ribbon, (2) polarity of the 
Ribbon, meaning light on dark or dark on light, (3) 
the width of the ribbon, meaning the distance be- 
tween the two lines, and (4) the orientation of the 
ribbon. 

13. The method of claim 12 wherein the comparing 
step comprises comparing predetermined reference 
graphs to the attributed sensed graphs based on the 
value that is a function of the difference between the 
degree of similarity between the reference and sensed 
graphs, and a predetermined threshold. 

14. The method of claim 13 wherein the comparing 
step utilizes a heuristically directed depth-first search 
technique to evaluate feasible matches between nodes 
and arcs of the attributed reference and sensed graphs. 

15. The method of claim 14 wherein the comparing 
step determines feasibility by the degree of match be- 
tween the node and arc attributes of each of the graphs, 
and a heuristic procedure is included to ignore paths 
that cannot produce a degree of similarity larger than 
the predetermined threshold. 

16. The method of claim 15 wherein the paths which 
lead to ambiguous solutions, comprising solutions that 
match a single object in one graph to multiple objects in 
the other, are ignored, and wherein paths that do not 
preserve the spatial relationships between the objects as 
they appeared in the image scene are ignored. 

***** 
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